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Abstract — The Space Shuttle program is ending and elements 
of the Constellation Program are either being cancelled or 
transitioned to new NASA exploration endeavors. The National 
Aeronautics and Space Administration (NASA) has worked 
diligently to select an optimum configuration for the Space 
Launch System (SLS), a heavy lift vehicle that will provide the 
foundation for future beyond low earth orbit (LEO) large-scale 
missions for the next several decades. Thus, multiple questions 
must be addressed: Which heavy lift vehicle will best allow 
the agency to achieve mission objectives in the most affordable 
and reliable manner? Which heavy lift vehicle will allow for a 
sufficiently flexible exploration campaign of the solar system? 
Which heavy lift vehicle configuration will allow for minimizing 
risk in design, test, build and operations? Which heavy lift 
vehicle configuration will be sustainable in changing political 
environments? 

Seeking to address these questions drove the development of an 
SLS decision-making framework. From Fall 2010 until Spring 
2011, this framework was formulated, tested, fully documented, 
and applied to multiple SLS vehicle concepts at NASA from 
previous exploration architecture studies. This was a multistep 
process that involved performing figure of merit (FOM)-based 
assessments, creating Pass/Fail gates based on draft threshold 
requirements, performing a margin-based assessment with sup- 
porting statistical analyses, and performing sensitivity analysis 
on each. This paper discusses the various methods of this 
process that allowed for competing concepts to be compared 
across a variety of launch vehicle metrics. The end result was 
the identification of SLS launch vehicle candidates that could 
successfully meet the threshold requirements in support of the 
SLS Mission Concept Review (MCR) milestone. 
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1. Introduction 

The decision for a new heavy-lift launch vehicle is rooted in 
the 2005 Exploration Systems Architecture Study (ESAS), 
which reinvigorated serious interest in a national heavy-lift 
capability. The Constellation Program (CxP), formed in 
response to the ESAS, established long duration exploration 
missions to the Moon and eventually Mars, which involved 
launching high-volume, large mass payloads into low earth 
orbit (LEO). The Ares V launch vehicle served the heavy 
lift need of the CxP and underwent multiple design concept 
studies and design analysis cycles between 2007 and 2010. 

As CxP was recommended for cancellation in early 2010, 
the National Aeronautics and Space Administration (NASA) 
conducted several internal studies on heavy-lift vehicle archi- 
tectures capable of performing a wide range of missions at 
reduced cost and schedule. The studies, most notably the 
Heavy-Lift Launch Vehicle (HLLV) study and the Heavy- 
Lift Propulsion Technology (HLPT) study, identified several 
families of launch vehicles whose features were distinguished 
by the number of stages, types of solid and liquid propulsion 
systems, and outer mold line (OML) design characteristics. 
These families included both side-mount and in-line config- 
urations, 27.6-foot and 33-foot OML diameters, 1.5 thru 3 
stages, multiple booster options, and a wide range of liquid 
propulsion engines and solid rocket motor options. 

The NASA Authorization Act of 2010 was passed by both 
the U.S. House of Representatives and U.S. Senate and was 
signed by the President on October 11, 2010. This Act 
directed NASA to develop a Space Launch System (SLS) 
capable of delivering crew and heavy cargo to LEO and 
beyond. As its initial capability, the SLS must be capable of 
lifting 70 metric tons of payload to LEO. A full capability of 
130 metric tons is achieved with the addition of a combined 
Upper and Earth Departure Stage to the initial configuration. 

With national stakeholder needs passed into law and various 
vehicle options to chose from, NASA George C. Marshall 
Space Llight Center (MSEC) established a team to develop 
a decision-making framework in which to communicate ef- 
fectively with agency decision-makers the characteristics and 
relative merits of the likely vehicle candidates. This frame- 
work will be summarized in this paper, with a specific focus 
on the methods themselves rather than the results produced. 
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2. Approach 

The team set out to establish a process by which the SLS 
Steering Committee could identify vehicle candidates that 
could meet documented NASA Mission Concept Review 
(MCR) success criteria. The process involved establishing 
a standardized set of process inputs, each of which have 
their own documented set of groundrules and assumptions. 
These standardized input sets and associated analytical pro- 
cesses were in the areas of performance, cost, and reliability 
estimation. By forming a common set of groundrules and 
assumptions for all discipline areas and associated teams, a 
consistent set of vehicle data was produced that would allow 
for relative comparisons to be made across many vehicle 
concepts. 

As with most complex problems, it is very difficult to define 
a single methodology or process that will identify a clearly 
attractive candidate from a multitude of options. Therefore, 
the decision team set out to define a multifactored process 
that would allow the problem to be assessed from a variety 
of angles. This process (as shown in Figure 1) includes the 
following steps: 

FOM-Based Assessment with figure of merit (FOM)s and 
criteria established from stakeholder metrics in the areas of 
performance, schedule, programmatics, and affordability. 
Draft SLS Level 1 Requirements assessment with all con- 
cepts evaluated on a pass or fail basis against the minimum 
threshold values across a variety of metrics. 

Aggregate Margin Assessment which compares all vehicle 
concepts to the draft Level 1 requirements and their margin 
with respect to each. This also includes supporting statistical 
analysis on the same metrics. 


3. FOM-Based Assessment 

Stakeholders in the SLS Decision 

In the heavy-lift vehicle decision process, the early estab- 
lishment of an agreed upon set of FOMs derived from the 
stakeholders in the decision is imperative. In this manner, the 
stakeholders are widely represented in the decision-making 
process, and the resulting vehicle decision is based on this 
representation rather than other external factors. Further- 
more, it provides a standard method for measuring and 
comparing competing concepts rather than independently 
assessing and potentially eliminating competitors without a 
larger- view of the trade environment. 

For NASA, the decision making process is fairly straightfor- 
ward. The NASA Administrator, the Office of Science and 
Technology Policy (OSTP), and the Office of Management 
and Budget (OMB) provides input to the President on existing 
and potential NASA programs. The President proposes new 
programs and/or the continuation of existing NASA programs 
in the annual Presidential Budget Request to Congress. The 
Congress debates the merits of the proposed programs and 
either chooses to authorize the programs or not. These 
authorized programs are then funded through the appropri- 
ations process. In the simplest case, the conglomeration of 
proposed, authorized, and appropriated programs are then 
passed and signed into law as the Nations Omnibus Spending 
Bill. This mutual agreement between the President and the 
Congressional Authorization and Appropriations processes 
are the general framework by which national decisions are 
made. 


Therefore, the Stakeholders for this decision are defined as 
the President of the United States, both the U.S. House of 
Representatives and U.S. Senate, and the NASA resources 
tasked with carrying out the authorized, approved, and funded 
program. 

Stakeholder Derived Metrics 

Determining the priorities of these stakeholders and the rela- 
tive importance of those priorities relative to one another is a 
more difficult task. The process used for this assessment was 
as follows: 

• Gather statements made by the stakeholders through their 
respective publications. 

• Bin those statements into general FOM categories. 

• Reduce similar statements into a single sub-FOM or what 
could potentially be termed heavy-life vehicle Needs, Goals, 
and Objectives (NGO). 

• Perform a first-order prioritization of those NGOs within 
the FOM categories. 

The first step of this process is merely collecting statements 
regarding the HLLV made by the President in the FY2011 
Presidential Budget Request and subsequent amendments 
thereof (speech at Kennedy Space Center (KSC) on April 
15th, 2010), as well as the Senate and House of Representa- 
tive Authorization Acts and Appropriations Reports. These 
statements were documented as “Quotes” associated with 
each source. 

For instance, individual quotes from these sources at the time 
of this study as it relates to the development schedule of the 
heavy-lift vehicle program are: 

President’s Speech at Kennedy Space Center on April 15th, 
2010: “And we will finalize a rocket design no later than 2015 
and then begin to build it.” 

U.S. House of Representatives NASA Authorization Act 
(HR 5781): “...the Administrator shall strive to meet the goal 
of having the heavy lift launch vehicle authorized in this 
paragraph available for operational missions by the end of 
the current decade” 

U.S. House of Representatives Appropriations Report: Not 
available at the time of this study 

U.S. Senate NASA Authorization Act (S.3729): “...Priority 
should be placed on the core elements with the goal for 
operational capability for the core elements not later than 
December 31, 2016.” 

U.S. Senate Appropriations Report: “...The Committee in- 
vests in a new heavy lift rocket to be built by 2017, along with 
the Orion capsule to carry astronauts, so NASA can again 
send humans on new journeys of discovery.” 

Organization and Simplification of Stakeholder Priorities 

The compilation of statements by the stakeholders was then 
organized in matrix format based on the general FOM areas 
that they targeted. These eventual general FOM areas found 
were Performance , Schedule , Pro grammatics, and Afford- 
ability aspects. Fortunately, several statements that were 
extracted were closely related with slight nuances between 
them. This led to a high level integration of closely related 
FOMs into what became known internally as “Integrated 
FOMs”. These Integrated FOMs, if moved forward into the 
environment of a NASA program, may be considered draft 
NGOs for the new Heavy-Lift Vehicle program. That is, 
they are NGOs by which vehicle level requirements may be 
derived from (e.g. integrated vehicle shall deliver 70 metric 
tons to LEO). 
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Figure 1 . Decision Analysis Process 


Once consolidated into the “Integrated FOMs” or NGOs, a 
high-level ranking of the NGOs was performed in order to 
reduce the quantity even further. A simple High, Medium, or 
Low priority was assigned to each NGO with an associated 
stakeholder that considers it to be of that priority. This 
technique was used to eliminate NGOs from the assessment 
which were medium or low in priority. This process resulted 
in all stakeholder priorities being distilled down into 16 
NGOs. Four of these were universally agreed to, three were 
unique to the Presidential priorities, five were unique to the 
Congressional priorities, and four were additional derived 
priorities. Figure 2 lists these NGOs. 

FOM Weightings 

With the four FOMs ( Performance , Schedule , Programmat- 
ics , and Affordability ) and the criteria that composes the 
FOMs established (Figure 2), the team proceeded to weight 
the FOMs. A commonly used technique to systematically 
work through complex decisions is the Analytical Hierarchy 
Process (AHP). This technique works by decomposing a par- 
ticular decision involving multiple variables into its individ- 
ual components and then comparing those two components 
independently of all other considerations. 

In this case, the individual FOMs are compared on a one- 
to-one basis to determine relative importance of one vs. the 
other. For instance, the four FOMs as listed are compared in 
an independent fashion systematically by answering six basic 
questions: 


Common Administration and Congress Values 

Performance 

Deliver a 70-100 t payload to LEO with Core Elements 

Schedule 

Perform first flight by the end of 2018 

Programmatic 

Maintain critical skills & transition workforce 

Afforability 

Minimize life-cycle costs (2030) 


Additional Administration Values 

Programmatic 

Provide a national advancement in technology 
Develop a national RP Engine 

Afforability 

Eliminate unnecessary infrastructure & right size industrial capacity 


Additional Congressional Values 

Performance 

Provide an evolved LEO capability of at least 130 metric tons with EDS 
Afforability 

Develop an initial capability under an $11. 5B cost cap through 2017 


Minimize cost risk 


Programmatic 

Utilize Shuttle/Ares Infrastructure/Hardware 


Decrease Loss of Crew Probability (an increase in LOC calculation) 


Additional Derived Values 

Afforability 

Proceed with a sustainable program (minimize annual recurring cost) 

Minimize peak cost 

Minimize cost to first flight 

Life Cycle Cost Reduction Challenges 


• Is Performance more important than Affordability? 

• Is Performance more important than Schedule? 

• Is Performance more important than Programmatics? 

• Is Affordability more important than Schedule 

• Is Affordability more important than Programmatics? 

• Is Schedule more important than Programmatics? 

These six questions are answered on a yes or no basis with 
a relative magnitude associated with that yes or no. A 
comparative matrix is composed and once completed, all 
values are normalized within each FOM category so that a 


Figure 2. Final Stakeholder Derived Priorities 


relative comparison can be made between them. This results 
in an AHP factor that demonstrates the relative strength in 
scoring of a particular factor over all others. The AHP factor 
itself can be used directly as a weighting factor in most 
circumstances. An easier to interpret method is to derive the 
percentage (out of a 100% total) that one factor holds over all 
others. This is merely the percentage of the AHP factor that 
one FOM accounts for divided by the total available AHP 
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factors. Through this AHP process, the FOM weightings for 
Performance , Schedule , Pro grammatics, and Affordability are 
found as shown in Figure 3 (based on a 100% total available): 


OUTPUT 

FOM Weight 


10% 

Performance 

55% 

Affordability 

25% 

Schedule 

10% 

Programmatics 


Figure 3. AHP Determined FOM Weightings 

NGO Scoring Criteria and Final Assessment 

Once the weighting of each FOM is found, it is imperative 
to set up a scoring method for each individual NGO. In 
this manner, data is measured using a consistently-applied 
process across a variety of vehicle concepts. For this as- 
sessment, NGO definitions (for the NGOs shown in Figure 
2) were agreed upon and documented. However, each NGO 
represents a certain type of data. For instance, performance 
represents a calculated payload mass to a certain orbit. 
Cost metrics represents a cost estimate of some sort for a 
given configuration, schedule and flight manifest. Schedule 
represents a development timeline found through a critical 
path assessment and so on. These aforementioned examples 
represent standardized, quantifiable data sources. In this 
assessment, there are also a multitude of data that represents 
subjective, hard-to-quantify NGOs. Examples of this data 
type include workforce transition, life-cycle cost reduction 
challenges, and others. Therefore, a standardized scoring 
methodology is established for these particular attributes. 

Additionally, it is required that all NGOs are measured ap- 
propriately. Two data conditioning steps must be undertaken 
to ensure a representative outcome: 

1) Ensure singular NGO quantities do not dominate the final 
consolidated measurement 

2) Ensure a large quantity of NGOs in a single FOM does 
not cause that FOM to dictate the results (regardless of the 
application of weighting factors) 

Whereas some NGOs are on the scale of tens or hundreds, 
other NGOs are on the scale of billions. It is apparent 
that a simple summation of these would result in the value 
represented by the lower scale having very little impact in 
the overall outcome. The method used to avoid this issue 
is a simple normalization between a fixed lower and upper 
boundary. For instance, performance is measured on the order 
of metric tons. A given range for performance may be a 
desired performance between 70 to 130 metric tons. By fixing 
a common lower boundary at the minimum in the range and 
a common upper boundary at the maximum of the range, all 
additional values can be interpolated between those two fixed 
points. If this process is repeated across all NGOs, then all 
are measured on a common scale. 

The second data conditioning technique used for this assess- 
ment is averaging within a single FOM category to avoid the 
quantity of NGOs within a single FOM from dictating the 
outcome. It is apparent that if all NGOs were merely summed 
for the final results, Affordability would dominate the final 
outcome based on the quantity of Affordability metrics within 
the assessment. Therefore, before application of any weight- 
ing factors, all Affordability NGOs are summed and divided 
by the quantity of Affordability NGOs (by stakeholder) to 


produce an average Affordability subtotal (by stakeholder and 
total). An important point about this technique is that this 
dictates that all NGOs are weighted equally within a single 
FOM. Once these data conditioning steps are applied, vehicle 
data can be imported and scoring comparisons can be made 
on any number of launch vehicle concepts. 


4. Draft SLS Requirements Assessment 

One particular shortfall of the FOM Based Assessment is that 
is does not capture how well the individual vehicles perform 
against the draft SFS Fevel 1 requirements as established 
by the SFS Steering Committee in November 2010. An 
assessment to these individual threshold values gives a much 
better indication of which vehicle concepts have the ability to 
meet standard NASA MCR success criteria based on eventual 
vehicle level requirements in all categories (cost, schedule, 
performance, reliability, etc.). In order to provide this insight, 
each individual requirement was given to the Requirements 
Analysis Cycle (RAC) teams as “Draft Threshold Require- 
ments” and “Draft Objective Requirements” across a variety 
of metrics. All of these threshold values, if levied as program 
requirements, must be met in order to be considered a feasible 
vehicle concept for SFS MCR. With this “Pass or Fail” 
criteria established, the vehicle data can then be applied to 
determine areas where the individual vehicle families fall 
short of these minimal “Threshold” values. Alternatively, 
vehicle concepts meeting most or all of these “Threshold” 
values should be considered feasible vehicles for the purposes 
of a MCR. This process is further depicted in Figure 4. 



Feasible Vehicle 
Options Identified 


Vehicles PASS that 
meet 90% of the 
threshold Level 1 
Vehicle Requirements 


Scoring 

Criteria 

Established 


First Pass Used to Identify Feasible Vehicle 
Concepts Based on Threshold Requirements 


Figure 4. MCR Feasible Vehicle Identification Process 


5. Aggregate Margin Study with 
Supporting Statistical Analysis 

System Margin Assessment 

At the inception of the SFS program, a measure of the 
“weighted system margin” is a very important metric that 
should be given appropriate thought when determining the 
preferred vehicle approach. While it is very difficult to 
quantify certain subjective aspects of the impending vehicle 
decision, the quantitative categories are easily binned accord- 
ing the level of “goodness” that each concept provides. These 
normalization bins give a relative measure of how much 
margin each vehicle concepts provides within the metrics 
established (with both the threshold and objective values 
considered). In addition, the weighting factors as currently 
understood can easily be applied to these quantitative metrics, 
resulting in a weighted system margin calculation. 
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Parametric Assessment of Vehicle FOMs 

To support the assessment of the vehicle parameter space and 
to enable the incorporation of uncertainty and constraints in 
the assessment methodology, a parametric and probabilistic 
Monte Carlo assessment of the RAC configurations was com- 
pleted. This assessment leveraged the work done by the RAC 
cycle and applied uncertainty on each of the FOMs for each 
of the vehicles. To complete the assessment, the following 
cases were assessed: 

1) Integrated vehicle assessment using multiple FOM 
weightings to assess sensitivity of the recommended optimal 
architecture to changes in FOM weighting 

2) Integrated vehicle assessment with sequential application 
of minimal requirements 

a) Performance (to standard orbit) 

b) Safety and Reliability (LOC/LOM) 

c) Cost (total development, near term, and recurring cost) 

d) Schedule (first flight) 

3) Integrated vehicle assessments with varying level of un- 
certainties on each of the FOMs 

The assessments were completed using reference vehicles 
that were defined using the draft SLS Level 1 requirements 
from the RAC process. Figure 5 depicts generic results of 
these Monte Carlo runs when implementing the Cost, Safety, 
and Performance gates. It can be interepreted as a probabilis- 
tic assessment of how many times out of all ran cases that the 
given configuration meets all levied requirement gates. The 
results from this assessment illustrates the sensitivity of the 
“optimal vehicle” to the applied constraints. 


Monte Carlo Count with Cost, Safety, and Performance Gates 



Vehicle 1 Vehicle 2 Vehicle 3 


Figure 5. Parametric Assessment Sample Results 


6. Conclusion 

The framework that was developed allowed for vehicle con- 
cepts and associated architectures to be compared on a variety 
of levels. These include the ability of the configurations to 
meet stakeholder needs, goals, and objectives, ability to meet 
SLS programmatic requirements (or requirement changes), 
and ability to provide margin to the program that will result in 
SLS having the ability to maintain both national and internal 
performance, schedule, and affordability targets. Combined, 
this multifactored assessment selected vehicles that would 
have the ability to successfully pass NASA MCR success 
criteria. Furthermore, the output of the framework was vetted 
by agency and national leadership. In this manner, it provided 


insight and confidence to agency decision-makers that the 
SLS concept chosen for further development will be a very 
flexible and capable vehicle that should serve the needs of a 
nation in its human space exploration endeavours for decades 
to come. 
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